bondscell_results$acdc5f8b-e79b-475f-95d8-f9623a705c9equeued¤logsrunning¦outputbodyMini: The Kullback-Leibler Divergencemimetext/htmlrootassigneelast_run_timestampA}*Gpersist_js_state·has_pluto_hook_features§cell_id$acdc5f8b-e79b-475f-95d8-f9623a705c9edepends_on_disabled_cells§runtime͵Tpublished_object_keysdepends_on_skipped_cells§errored$371e4d9b-99d9-412b-8648-b19f9bfad216queued¤logsrunning¦outputbody

Statistics of the distribution $p$ that we keep constant, the mean of $q$ is interactive. Feel free to edit the code to change more parameters.

mimetext/htmlrootassigneelast_run_timestampA}*Sgpersist_js_state·has_pluto_hook_features§cell_id$371e4d9b-99d9-412b-8648-b19f9bfad216depends_on_disabled_cells§runtimeTpublished_object_keysdepends_on_skipped_cells§errored$8f1a66b0-6a52-4eba-bc1f-0dd99eb91cb9queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA}*݌;persist_js_state·has_pluto_hook_features§cell_id$8f1a66b0-6a52-4eba-bc1f-0dd99eb91cb9depends_on_disabled_cells§runtimeΜ&Opublished_object_keysdepends_on_skipped_cells§errored$9834138c-3370-4ba5-a28c-8d74f90869caqueued¤logsrunning¦outputbody^mimetext/htmlrootassigneelast_run_timestampA}*fpersist_js_state·has_pluto_hook_features§cell_id$9834138c-3370-4ba5-a28c-8d74f90869cadepends_on_disabled_cells§runtimeX/ڥpublished_object_keysdepends_on_skipped_cells§errored$3c655528-77f5-4327-9a0b-1e0a7a27474bqueued¤logsrunning¦outputbodyٯ

Interactive Example

Here is an animation that shows the KL divergence between two Gaussian distributions:

mimetext/htmlrootassigneelast_run_timestampA}*Sg[persist_js_state·has_pluto_hook_features§cell_id$3c655528-77f5-4327-9a0b-1e0a7a27474bdepends_on_disabled_cells§runtime ߵpublished_object_keysdepends_on_skipped_cells§errored$62694615-197c-4c2c-8365-c1f6cb974564queued¤logsrunning¦outputbody mimeimage/svg+xmlrootassigneelast_run_timestampA}*'\persist_js_state·has_pluto_hook_features§cell_id$62694615-197c-4c2c-8365-c1f6cb974564depends_on_disabled_cells§runtimed4published_object_keysdepends_on_skipped_cells§errored$1f8d33a3-aeda-45a7-a343-ab3565ace9aequeued¤logsrunning¦outputbodyprefixFloat64elements12.5text/plain12.005text/plain11.52text/plain11.045text/plain10.58text/plain10.125text/plain9.68text/plain9.245text/plain 8.82text/plain 8.405text/plain 8.0text/plain 7.605text/plain 7.22text/plain6.845text/plain6.48text/plain6.125text/plain5.78text/plain5.445text/plain5.12text/plain4.805text/plainmore\8.405text/plain]8.82text/plain^9.245text/plain_9.68text/plain`10.125text/plaina10.58text/plainb11.045text/plainc11.52text/plaind12.005text/plaine12.5text/plaintypeArrayprefix_shortobjectid2510615373412cbdmime!application/vnd.pluto.tree+objectrootassigneekllast_run_timestampA}*X?persist_js_state·has_pluto_hook_features§cell_id$1f8d33a3-aeda-45a7-a343-ab3565ace9aedepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$ea709a1e-ae4a-4306-8ed6-656a90b1e014queued¤logsrunning¦outputbodyN

Behind the scenes

mimetext/htmlrootassigneelast_run_timestampA}*Sh6persist_js_state·has_pluto_hook_features§cell_id$ea709a1e-ae4a-4306-8ed6-656a90b1e014depends_on_disabled_cells§runtimeErpublished_object_keysdepends_on_skipped_cells§errored$4d4e9bb3-41ef-4185-9930-1628220c6552queued¤logsrunning¦outputbody1kullback_leibler (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA}*]Jpersist_js_state·has_pluto_hook_features§cell_id$4d4e9bb3-41ef-4185-9930-1628220c6552depends_on_disabled_cells§runtime&published_object_keysdepends_on_skipped_cells§errored$c9702675-e665-49ff-a86a-47b88d7269cbqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA}*Sh=persist_js_state·has_pluto_hook_features§cell_id$c9702675-e665-49ff-a86a-47b88d7269cbdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$ec537d0f-b127-4512-a743-f374fed7e2a2queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA}*epersist_js_state·has_pluto_hook_features§cell_id$ec537d0f-b127-4512-a743-f374fed7e2a2depends_on_disabled_cells§runtimeC5published_object_keysdepends_on_skipped_cells§errored$2e9f5142-67fd-4098-90b1-f75f05565d98queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA}*lpersist_js_state·has_pluto_hook_features§cell_id$2e9f5142-67fd-4098-90b1-f75f05565d98depends_on_disabled_cells§runtimeOĵpublished_object_keysdepends_on_skipped_cells§errored$f715e333-a787-40ff-99d9-a2bf75385315queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA}*0persist_js_state·has_pluto_hook_features§cell_id$f715e333-a787-40ff-99d9-a2bf75385315depends_on_disabled_cells§runtimeOٵpublished_object_keysdepends_on_skipped_cells§errored$e39fd1e2-cbba-48ad-a5f7-733a229be257queued¤logsrunning¦outputbody

The Kullback-Leibler Divergence (a.k.a. relative entropy) between two distributions $q$ and $p$ is defined as

$$D_{\text{KL}}[q,p] \equiv \sum_z q(z) \log \frac{q(z)}{p(z)}$$

The following Gibbs Inequality holds (see wikipedia for proof):

$$D_{\text{KL}}[q,p] \geq 0 \quad \text{with equality only if } p=q $$

The KL divergence can be interpreted as a distance between two probability distributions.

As an aside, note that $D_{\text{KL}}[q,p] \neq D_{\text{KL}}[p,q]$. Both divergences are relevant.

mimetext/htmlrootassigneelast_run_timestampA}*Kgpersist_js_state·has_pluto_hook_features§cell_id$e39fd1e2-cbba-48ad-a5f7-733a229be257depends_on_disabled_cells§runtime s5cpublished_object_keysdepends_on_skipped_cells§errored±cell_dependencies$acdc5f8b-e79b-475f-95d8-f9623a705c9eprecedence_heuristic cell_id$acdc5f8b-e79b-475f-95d8-f9623a705c9edownstream_cells_mapupstream_cells_maptitle$371e4d9b-99d9-412b-8648-b19f9bfad216precedence_heuristic cell_id$371e4d9b-99d9-412b-8648-b19f9bfad216downstream_cells_mapupstream_cells_map@md_strgetindex$8f1a66b0-6a52-4eba-bc1f-0dd99eb91cb9precedence_heuristiccell_id$8f1a66b0-6a52-4eba-bc1f-0dd99eb91cb9downstream_cells_mapDistributionsPlotsBmlipTeachingToolsLaTeXStrings$62694615-197c-4c2c-8365-c1f6cb974564StatsPlotsPlotMeasuresupstream_cells_map$9834138c-3370-4ba5-a28c-8d74f90869caprecedence_heuristic cell_id$9834138c-3370-4ba5-a28c-8d74f90869cadownstream_cells_mapμ_q$f715e333-a787-40ff-99d9-a2bf75385315$62694615-197c-4c2c-8365-c1f6cb974564#474#bondupstream_cells_map#PlutoRunner.PlutoRunner.create_bondHypertextLiteral.BypassHypertextLiteral.contentSlider@bindnameμ_seq$2e9f5142-67fd-4098-90b1-f75f05565d98PlutoRunnerPlutoRunner.Base.getHypertextLiteral.ResultHypertextLiteralPlutoRunner.Core.applicable$3c655528-77f5-4327-9a0b-1e0a7a27474bprecedence_heuristic cell_id$3c655528-77f5-4327-9a0b-1e0a7a27474bdownstream_cells_mapupstream_cells_map@md_strgetindex$62694615-197c-4c2c-8365-c1f6cb974564precedence_heuristic cell_id$62694615-197c-4c2c-8365-c1f6cb974564downstream_cells_mapupstream_cells_mapstringLaTeXStrings.latexstringislesskullback_leibler$4d4e9bb3-41ef-4185-9930-1628220c6552Kullback-Leibler Divergence")) (a.k.a. relative entropy) between two distributions ``q`` and ``p`` is defined as ```math D_{\text{KL}}[q,p] \equiv \sum_z q(z) \log \frac{q(z)}{p(z)} ``` The following [Gibbs Inequality](https://en.wikipedia.org/wiki/Gibbs%27_inequality) holds (see [wikipedia](https://en.wikipedia.org/wiki/Gibbs%27_inequality) for proof): ```math D_{\text{KL}}[q,p] \geq 0 \quad \text{with equality only if } p=q ``` The KL divergence can be interpreted as a distance between two probability distributions. As an aside, note that ``D_{\text{KL}}[q,p] \neq D_{\text{KL}}[p,q]``. Both divergences are relevant. """metadatashow_logsèdisabled®skip_as_script«code_foldedënotebook_id$ef07c3b4-4560-11f1-b0bb-7bc9e1ff1064in_temp_dir¨metadatafrontmatterimagehttps://imgur.com/DdKOZVv.pngtitle%Mini: The Kullback-Leibler DivergencedescriptionLIntroduction to the Kullback-Leibler Divergence with an interactive example.